1. Field of the Invention
The present invention relates generally to an image processing and modulation system for a digital display system, and more particularly to performing lighting control along with advanced image processing and image modulation to produce high-quality output images.
2. Discussion of Prior Art
The many types of Flat Panel Display (FPD) technologies include Active Matrix Liquid Crystal Displays (AMLCDs) also referred to as Thin Film Transistor displays (TFTs), silicon Reflective LCDs (si-RLCDs), Liquid Crystal On Silicon (LCOS), ferroelectric displays (FLCs), Field Emission Displays (FEDs), Carbon Nanotube based Nano Emissive Displays (NEDs), ElectroLuminescent Displays (ELDs), Light Emitting Diodes (LEDs), Organic LEDs (OLEDs), Plasma Displays (PDs), Passive matrix Liquid Crystal Displays (LCDs), Thin Film Transistor (TFT), Silicon X-tal Reflective Displays (SXRD) and Digital Mirror Displays (DMDs).
Manufacturing FPDs requires special processing steps and it is often difficult to achieve acceptable visual quality in terms of consistence across the entire area of the display. Besides being a yield issue for new displays, display characteristics can change over operating conditions and over the lifetime of the display. Most FPD technologies also require light sources, such as fluorescent lamps, Cold Cathode Florescent lamps, Ultra High Power (UHP) lamps or Light Emitting Diodes (LEDs) to illuminate the display images. LEDs may be made from a variety of materials and each may have their own “color temperature” characteristics. Other systems may use a hybrid of a traditional lamp along with one or more white or colored LED arrays. In one example system, Sharp Corporation of Japan uses a cold cathode florescent lamp along with red LEDs in a hybrid back light approach for 37″ and 57″ LCD TVs.
Some of these technologies can be viewed directly as FPDs and others are used as microdisplay devices where the image is projected and a user views the projected image.
Projection display systems may use transmissive or reflective “microdisplay” technologies. To achieve the desired full color gamut in microdisplay based parallel color projection systems, three separate LCD image modulators, one for each of the R, G, and B (RGB) components, may be used. A single LCD image modulator which produces R, G, and B, either through spatial color filters or with sequential color fields at a sufficiently high rate can cost less. Other light sources such as LEDs and lasers may also be used for the different colors with one, three or more imagers. Reflective technologies such as Texas Instruments Digital Light Processing (TI DLP) and technologies such as LCOS imagers are other examples of popular microdisplay technologies. Sony uses a Silicon X-tal Reflective Display (SXRD) microdisplay in some of their rear projection television systems. Other display devices may be based on High Temperature Poly-Silicon (HTPS) technology, Micro Electro-Mechanical Systems (MEMS) or Carbon Nanotubes.
Projection displays have a projection path with associated additional properties. For example, projecting an image from a projector on a level below the middle of a flat screen results in an image which is narrower at the bottom than at the top in what is known as the “Keystone” effect. Radial distortion occurs when an image pixel is displaced from its ideal position along a radial axis of the image. Because an image has the largest field angles in the display corners, the corners exhibit worse radial distortion than other display areas. Radial distortion includes barrel distortion, where image magnification decreases towards the corners, and pin cushion distortion, where the magnification increases towards the corners. Lens related distortions including radial distortion can cause image deformation. Distortion can also result from non-flat screens or the Earth's magnetic field. Shortening the projection path, often done to thin a rear projection system, also increases distortion. The non-uniform projection characteristics of a display system may be represented by a “distortion map” which is based on a single pixel or a group of pixels. A “tessellation map” for characterizing the display distortion may be made up of non-uniform triangles where the mesh of triangles is denser where the amount of distortion is highest.
Image modulators have a fixed number of pixels spaced uniformly in a pattern. Projecting an image from an image modulator to a display screen deforms the uniformity of pixel spacing. In other words, pixels are not correlated one-to-one from a sub-area of the image modulator to the corresponding sub-area of the display screen. Therefore, some screen display areas have more image modulator pixels than screen pixels while other screen display areas have fewer image modulator pixels than screen pixels.
For panoramic displays, motion artifacts appear where image objects move near the edges of curved screens. Even when a flat screen projection is motion-adaptive filtered, the difference in the distances of objects from the projector causes an apparent motion of moving objects on a curved screen. Additionally, extremely large curved screen systems can achieve necessary resolution and brightness only with multiple light sources. Even very large flat screens typically require multiple light sources to accurately control brightness and color. Control of multiple light sources becomes a critical factor in achieving high quality images for large screen format displays.
During the production of content, multiple camera systems are commonly used to improve display quality on curved screen and very large screen displays. For example, two cameras record overlapping halves of a scene to increase the content of the scene being captured. The increase of content results from capturing a wider scene and different angles of the same scene. A layered coding technique may include a standard codec stream as a base layer and enhancement information as a supplemental layer. Even if the two views are from slightly different angles, the compression ratio for the two camera views combined is less than the total compression ratio would be if each view were captured and compressed independently. Additionally, the second camera can provide a view that may be occluded from the first camera.
Systems using additional camera angles for different views can provide additional coded and compressed data for later use. Multiple camera systems can also compensate for the limited focal depth of a single camera and can substitute for the use of a depth-finding sensor which senses and records depth information for scenes. Multiple cameras can also be used to allow a more interactive playback environment where the user can either choose the angle from which to watch a scene, zoom in or out of a scene, or as a result of game play or similar controls, be shown a different view of a scene.
Stereoscopic video capture also uses multi-camera systems in which a first camera records a left-eye view and a second camera records a right-eye view. Because camera lenses focus at a certain distance, one camera uses one focal plane for all objects in a scene. A multi-camera system can use multiple cameras each to capture a different focal plane of a single scene. This effectively increases the focal depth. Digital image processing can further improve focusing for these multi-camera systems.
Types of three dimensional binocular display systems include anaglyph displays, frame sequence displays, autostereoscopic displays, single and multi-turn helix displays. These normally have multiple camera data channels. Anaglyph systems usually require a user to wear red and green glasses so that each eye perceives a different view. Frame sequencing systems use shutter glasses to separate left and right views. Autostereoscopic displays use lenticular lenses and holographic optical elements. Single or multi-turn helix displays use multiple semi-transparent display screens which can be seen by multiple observers without special glasses.
Multiple camera systems can benefit from image processing during both the capture and the playback of the data. During the capture, the different camera images can be registered to each other to create a combined video. The registration gives a way to relate the video from the cameras to each other and to the viewer. In a more sophisticated capture system, the various video images can be captured into a 3D environment where a 3D triangle map of the video images is used to make a video database. During playback, the view seen by the user can be manipulated and, by using the multiple video registration or the 3D triangle map, can be manipulated to give the user the best view. Since the user viewpoint at times will be unique compared to the captured video, the display output system can manipulate the video data via geometric transforms to produce a new view for the user which was not one of the captured views.
Because of inherent response characteristics of LCD materials, LCD-based projection and direct view display systems each have unique flicker characteristics and exhibit different motion artifacts. Further, LCDs switch in the finite time it takes to change the state of a pixel. Active matrix thin film transistor (TFT) displays, which have an active transistor controlling each display pixel, require a switching time related to the LCD material composition and thickness, and to the techniques of switching. The transitions from one state to another are also not necessarily linear and can vary depending on the sequence of pixel values.
The ability to accurately portray each color component for each pixel at each screen location is another desired function of a high quality display device. Since a display output pixel typically consists of a triad (red, green, blue) or other combination of color components, each color component for each pixel is typically made up of multiple sub-pixels. However, the term “pixel” is also used to describe the triad of sub-pixels. Each color component of the pixels mapped to the screen display pixels may transfer non uniformly where some pixels end up too bright and others end up not bright enough. This non-uniformity may be a function of the light path characteristics, the mirrors, the lenses, the lamps or any combination thereof Similarly for LED backlit FPDs, the spacing and positioning of the LEDs may cause non-uniform brightness patterns where the image modulator pixels closer to the LEDs appear brighter than the image modulator pixels further from the LEDs. In the case of colored LEDs or LEDs with color filters, the modulator pixels may be further affected not just with brightness differences, but with color uniformity differences. The brightness of LEDs can be modulated using various Pulse Width Modulation (PWM) techniques or varied by adjusting the voltage or current. The LEDs may be controlled collectively, by proximity in groups, by color, by some combination or in the most sophisticated case each LED can be controlled individually.
Display systems typically include various user controls for color, brightness, white balance (color temperature) and other settings. Users adjust the controls based on preference, ambient lighting conditions, and type of content (video versus computer). Over time, lifetime effects of the display may require the user to make further adjustments or the FPD may automatically make such adjustments. The user adjustments are typically on the entire system, not on subsections of the display, though factory trained technicians using instruments could perform other more sophisticated adjustments. Some display systems include capabilities to measure light output feedback from both lighting sources as well as the ambient lighting conditions. Such a feedback system could be used to compensate for temperature and lifetime effects of the display components and the changing ambient light conditions. The timescale for such adjustments will vary based on the specific measurements and the criticality of the adjustments. Adjustments based on the type of content may be made in multiple seconds, while frame to frame adjustments for dynamic control need to be done much faster, on the order of milliseconds. Adjustments based on ambient light conditions may naturally vary over the time of day as well as with changes to the lamps and other light sources in a room.
FIG. 1A shows a front view of an LED array 102f made up of a structure of multiple LEDs 104. Each LED is a light source which can be either a white LED or one of a variety of colors such as red, green or blue. Other LED colors are also used and the color may be part of the LED structure, may be a color filter as part of each LED or may be a filter mechanism for the array of LEDs. The array of LEDs may be arranged in a pattern of colors so as to produce a controllable and consistent light source that, when combined with an image modulator, is capable of producing the full gamut of display colors.
FIG. 1B shows subsystem 100 which is a side view of an LED array 102s structured from multiple LEDs 104, a multi-lens system 108 and 112 and a microdisplay imager or image modulator 110 that may use a variety of light modulation techniques. The LEDs may be white or multicolored and an additional color system (not shown), such as a color wheel, may also be included in system 100. In one example system, lens 108 concentrates the light through the microdisplay 110, and lens 112 controls the light projection. The projection system may be front or rear projection and use other lenses and mirrors, dichroic recombiners and filters to project the image on a screen (not shown). Microdisplay 110 may be either transmissive, passing modulated amounts of light through each pixel, or reflective, reflecting modulated amounts. The resolution of the microdisplay 110 is typically much higher than the resolution of the components that make up the light source.
FIG. 1C shows subsystem 140 which is a side view of an LCD panel system where a backlight 122 is made up of either strips 122 or an array 102 of LEDs. The LCD display or image modulator 130 typically has a much higher resolution than the resolution of the components that make up the light source. Exceptions include OLEDs, where each LCD pixel has a light source as part of the pixel and as such the resolution of the light sources is the same as that of the display and NEDs, where carbon nanotubes are arranged such that a properly applied voltage excites each nanotube which in turn bombards the color phosphors for the pixels or subpixels with electrons and “lights them up.” The brightness and color wavelength is based on the quantity of electrons, the phosphors used and a variety of other factors.
For LED backlit displays, various filters, color and light diffusion gradients, Brightness Enhancement Films (BEF), diffusion plates, light guides and mixing light guides 126 may be used as an option to improve the light uniformity for the display. Some backlit displays may also include one or more optical sensors (not shown in FIG. 1) that can detect changes in brightness and wavelength (see Hamamatsu product brief).
Most LCD-based image modulators (110 and 130) are addressed in raster scan fashion and each pixel is refreshed during each display frame interval. Accordingly, every output pixel is written to the display during every refresh cycle regardless of whether the value of the pixel has changed since the last cycle. Each R, G, and B color component normally has a different intensity value which is digitally represented by a number of bits. For example, if 8 bits represent each R, G, and B color component, then each component has 28 (=256) intensity values from 0 to 255. Changing the intensity value of a color component in an ideal digital device from a number X, for example, to a number Y, takes just as long regardless of the Y value. So in an ideal system, changing a color component value from 2 to 3 takes as long as changing the value from 2 to 200. However, because of the nature of LCD image modulator pixels, the transitions for modulating light intensities are not purely digital, and are not necessarily linear. U.S. Pat. No. 6,340,994 by Margulis et al. teaches Temporal Gamma Processing (TGP) which assures that the time-related representations of an image are as accurate as possible, , based on a previous frame value and a known transfer function of the display modulation system and adjusting its output to a desired value during display of a desired frame.
The light sources for a modulator based system may not be able to accurately and uniformly reproduce the intended color gamut, and different modulator pixel positions on the screen may be affected by the light sources differently, causing a display of non-uniform color and brightness. Additionally, the light sources' output characteristics may change over time and need to be compensated for. The display system may include appropriate sensors to determine the status of the light sources.
Therefore, for all the foregoing reasons, what is needed is an image processing system to effectively enhance display quality by controlling both the image modulator and the light sources and thereby provide better visual images.